Exercise 7.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 7\n",
    "\n",
    "We saw that having a prior $p(\\mathbf{w}) \\propto \\exp(-\\alpha \\sum_j w_j^2)$ yielded Ridge regression as a maximum likelihood estimator (MLE).  Specifically: \n",
    "If\n",
    "$$p(\\mathbf{w}) \\propto \\exp(-\\alpha \\sum_j w_j^2)$$\n",
    "Then $p(\\mathbf{w} | X, \\mathbf{y})$ is maximized when\n",
    "$$\n",
    "C(\\mathbf{w}) = \\sum_j (\\mathbf{x}_j\\cdot\\mathbf{w} - y_j)^2 + \\alpha' \\sum_j w_j^2\n",
    "$$\n",
    "\n",
    "is minimized.\n",
    "\n",
    "1. What is the MLE loss function $C(\\mathbf{w})$ if $p(\\mathbf{w}) \\propto \\exp(-A\\sum_j w_j^3)$?\n",
    "2. What should $p(\\mathbf{w})$ look like to recover Lasso regression as an MLE?\n",
    "3. (Bonus) Note that $\\alpha$ in the prior equation is different from $\\alpha'$ in the MLE equation.  Work out precisely the value of $\\alpha'$ in the second equation in terms of $\\sigma$ (the standard deviation of the data) and $\\sigma_w$ (the standard deviation of the prior).  It helps to know that \n",
    "$$\\mathcal{N}(\\mathbf{x}; 0, \\sigma) \\propto \\exp \\left( -\\frac{\\sum_j x_j^2}{2\\sigma^2}\\right),$$\n",
    "and to keep in mind that minimizing $k \\cdot C(\\mathbf{w})$ is the same as minimizing $C(\\mathbf{w})$, so long as $k > 0$.  "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pydata_nyc20173.6",
   "language": "python",
   "name": "pydata_nyc20173_6"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}